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- T/ie MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )^ Responsive to communication(s) filed on 03 May 2004 . 
2a)S This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-25 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) |EI Claim(s) 1-25 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 1 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12)D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. Q Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 

1 . This office action is responsive to the following communications: 
Application, filed 03/27/2001 ; Amendment A, filed 06/30/2003; Amendment B, 
filed 1 1/10/2003; Amendment C, filed 12/15/2003; Amendment D, filed 
05/03/2004. 

This action is made final. 

2. Claims 1-24 are pending in this application. Claims 1 and 13 are 
independent claims. Claim 25 has been added. 

3. The present title of this application is "Augmented Reality Guided 
Instrument Positioning with Modulated Guiding Graphics" (as originally filed). 

Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for 
all obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described 
as set forth in section 102 of this title, if the differences between the subject matter sought to 
be patented and the prior art are such that the subject matter as a whole would have been 
obvious at the time the invention was made to a person having ordinary skill in the art to which 
said subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 

5. Claims 1-24 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Fukunaga, U.S. Patent Number 6,346,940 B1, "Virtualized Endoscope 
System", class 345/427, 02/2002, filed 02/1998. 

Regarding claim 1, Fukunaga discloses a method for augmented reality 
guided instrument positioning (col. 2, lines 11-20), comprising the steps of: 
displaying a real view of an environment (col. 5, lines 34-36) determining a 
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graphics guide for positioning an instrument; (col. 5, lines 35-39) and augmenting 
the real view with a rendering of the graphics guide (col. 5, lines 30-46) such that 
a transparency of at least one portion of the graphics guide is modulated (col. 7, 
lines 59-64) 

However, it is noted that Fukunaga fails to disclose transparency in and 
out of an augmented view repeatedly over a predefined time interval, 
independent of a change in a view of the graphics guide 

Fukunaga disclose the semi transparency formation unit thins pixels 
according to conditions that include an adjustable degree of transparency and 
displaying a transparent endoscope image. 

It would have been obvious to one of ordinary skill in the art at the time of 
the invention of Fukunaga to include transparency of an augmented view of the 
graphics guide including modulating the guide in and out of a view independent 
of a change in view, because Fukunaga disclose the semi transparency is 
prepared according to conditions and the conditions may include a modulated 
transparency, at a desired position on the image, as the virtual endoscope is 
moved about the image to view the represented portion of the image the 
endoscope is occupying, with and without the guide displayed. 

Regarding claim 2, Fukunaga discloses rendering step comprises the step 
of varying the transparency of the at least one portion of the graphics guide with 
respect to other portions of the graphics guide to provide a substantially 
unobstructed view through the at least one portion of the graphics guide . . . (col. 
7, lines 40-46) 
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However, it is noted that Fukunaga fails to disclose a transparent guide 
marker. I 

It would have been obvious to one of ordinary skill in the art at the time of 
the invention to include not only semi-transparent images, but also markers to 
avoid obstructing portions of the image, which is what Fukunaga wishes to 
accomplish. 

Regarding claim 3, Fukunaga discloses varying the transparency of the at 
least one portion of the graphics guide during the pre-defined time intervals to 
provide a substantially unobstructed view through the at least one portion of the 
graphics guide . . . (col. 7, lines 59-64) 

However, it is noted that Fukunaga fails to disclose a transparent guide 
marker. 

It would have been obvious to one of ordinary skill in the art at the time of 
the invention to include not only semi-transparent images, but also markers to 
avoid obstructing portions of the image, which is what Fukunaga wishes to 
accomplish. 

Regarding claim 4, Fukunaga discloses varying the transparency of each 
of a plurality of portions of the graphics guide during at least one predefined time 
interval to provide a substantially unobstructed view through each of the plurality 
of portion to at least a portion of the instrument . . . 

Fukunaga discloses an adjustable degree of transparency, col. 7, lines 59- 

64. 
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However, it is noted that Fukunaga fails to disclose a transparent guide 
marker. 

It would have been obvious to one of ordinary skill in the art at the time of 
the invention to include not only semi-transparent images, but also markers to 
avoid obstructing portions of the image. 

Regarding claim 5, Fukunaga discloses wherein plurality of portions is 
consecutive, (col. 8, lines 5-10) 

Regarding claim 6, Fukunaga discloses varying the transparency of the at 
least one portion of the graphics guide such that the at least one portion 
repeatedly switches between transparent and less transparent, (thinning pixels 
based on a preset degree of semi transparency adjusted by the operator, col. 1 1 , 
lines 44-57) 

However, it is noted that Fukunaga fails to disclose a transparent guide 
marker. 

It would have been obvious to one of ordinary skill in the art at the time of 
the invention to include not only semi-transparent images, but also guide markers 
to avoid obstructing portions of the image. 

Regarding claim 7, Fukunaga discloses constructing the graphics guide as 
a line, and rendering step comprises the step of modulating the transparency of 
the line with respect to time so that the line repeatedly fades in and out of view to 
provide a substantially unobstructed view . . . (the guiding marker used to 
indicate the direction could be a path, arrows, etc, col. 8, lines 1-10) 

However, it is noted that Fukunaga fails to disclose a transparent line. 
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It would have been obvious to one of ordinary skill in the art at the time of 
the invention to include not only semi-transparent images, but also markers to 
avoid obstructing portions of the image. 

Regarding claim 8, Fukunaga discloses constructing the graphics guide as 
a line, and rendering step comprises the step of modulating the transparency of 
portions of the line so that at least a portion of the instrument is substantially 
unobstructed . . . (the guiding marker used to indicate the direction could be a 
path, arrows, etc, col. 8, lines 1-10) 

However, it is noted that Fukunaga fails to disclose a transparent line. 

It would have been obvious to one of ordinary skill in the art at the time of 
the invention to include not only semi-transparent images, but also markers to 
avoid obstructing portions of the image. 

Regarding claim 9, Fukunaga discloses rendering step comprises the step 
of modulating the transparency of portions of the line with respect to time and 
space so that at least a portion of the instrument is substantially unobstructed . . . 
during pre-defined time intervals, (semitransparent image so that portions of the 
image can be viewed unobstructed, col. 1 1 , lines 44-67) 

However, it is noted that Fukunaga fails to disclose a transparent line. 

It would have been obvious to one of ordinary skill in the art at the time of 
the invention to include not only semi-transparent images, but also markers to 
avoid obstructing portions of the image. 

Regarding claim 10, Fukunaga discloses constructing the graphics guide 
as a cylinder . . . rendering step comprises the step of modulating the 
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transparency of the cylinder with respect to time so that the cylinder repeatedly 
fades in and out of view . . . 

Fukunaga discloses typical examples of guiding markers. 

However, it is noted that Fukunaga fails to disclose a transparent cylinder. 

It would have been obvious to one of ordinary skill in the art at the time of 
the invention to include not only semi-transparent images, but also graphic 
markers to avoid obstructing portions of the image. 

Regarding claim 1 1 , Fukunaga discloses modulating the transparency of 
portions of the cylinder so that at least a portion of the instrument is substantially 
unobstructed . . . (semitransparent image sot that portions of the image can be 
viewed unobstructed, col. 11, lines 44) 

However, it is noted that Fukunaga fails to disclose a transparent cylinder. 

It would have been obvious to one of ordinary skill in the art at the time of 
the invention to include not only semi-transparent images, but also graphic guide 
markers to avoid obstructing portions of the image. 

Regarding claim 12, Fukunaga discloses rendering step comprises the 
step of modulating the transparency of portions of the cylinder with respect to 
time and space so that at least a portion of the instrument is substantially 
unobstructed . . . during pre-defined time intervals, (an operation path history that 
registers the path position and view direction each times the instrument changes 
to reproduce the image on the display unit, col. 8, lines 32-47) 

However, it is noted that Fukunaga fails to disclose a transparent line. 
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It would have been obvious to one of ordinary skill in the art at the time of 
the invention to include not only semi-transparent images, but graphic also guide 
markers to avoid obstructing portions of the image. 

Regarding claims 13-24, they are rejected based upon similar rational as 
above claims 1-12 respectively. 

However, it is noted that Fukunaga fails to disclose a video camera 
providing a real view. 

Fukunaga discloses the endoscope system provides real images on the 
monitor, col. 5, lines 30-35, and the system includes plural input devices, col. 29, 
lines 1-13. 

It would have been obvious to include a video camera as an input device 
to include all forms of input for image processing, because video cameras are 
well know in the art as an apparatus for providing real images. 

Regarding claim 25, it is rejected based upon similar rational as above 
claim 2. 

Regarding claims 26 and 27, Fukunaga fails to disclose wherein the real 
view is a contemporaneous view from a camera of the instrument. 

Fukunaga discloses a command to display body position direction 
compass image including a view of the body relative to the instrument, i.e. 
endoscope tip, col. 14, lines 35-67) 

It would have been obvious to include a view from a camera to the 
instrument to include the display corresponding to a desired visual field from a 
desired viewpoint position, as disclosed in Fukunaga. 
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Response to Arguments 

6. Applicant's arguments with respect to claims 1-25 have been considered 
but are moot in view of the new ground(s) of rejection. 



Conclusion 

7. Applicant's amendment necessitated the new ground(s) of rejection 
presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. 
See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as 
set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire 
THREE MONTHS from the mailing date of this action. In the event a first reply is 
filed within TWO MONTHS of the mailing date of this final action and the advisory 
action is not mailed until after the end of the THREE-MONTH shortened statutory 
period, then the shortened statutory period will expire on the date the advisory 
action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be 
calculated from the mailing date of the advisory action. In no event, however, will 
the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 



Any inquiry concerning this communication or earlier communications from 
the examiner should be directed to Motilewa A. Good-Johnson whose telephone 
number is (703) 305-3939. The examiner can normally be reached on Monday - 
Friday 8:30 AM - 5:00 PM. 
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If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, Mike Razavi can be reached on (703) 305-4713. The fax 
phone number for the organization where this application or proceeding is 
assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from 
the Patent Application Information Retrieval (PAIR) system. Status information 
for published applications may be obtained from either Private PAIR or Public 
PAIR. Status information for unpublished applications is available through 
Private PAIR only. For more information about the PAIR system, see http://pair- 
direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll- 
free). 
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Examiner 
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